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Today, the spread of Internet use has accelerated the development of educational technologies and 
increased the quality of education by encouraging teachers’ cooperation and participation. As a result, 
examinations executed via the Internet have become common, and a number of universities have 
started using distant education management system. Eventually, today, more people have a chance to 
take education. Measurement and evaluation applications carried out via the Internet are now quite 
important for education. In the present study, a valid and reliable attitude scale was developed to 
measure the attitudes of students doing the Distant Education Theology Undergraduate Education 
Program towards a web-based examination system (MOODLE).The study group was made up of 1287 
3rd and 4th grade students registered in the Distant Education Application and Research Center. In the 
study, exploratory factor analysis (EFA) was conducted to determine the factor structure of the scale; 
confirmatory factor analysis (CFA) was conducted to test its construct validity; and other validity 
analyses such as exploratory factor analysis, uni-dimensional factor analysis and two-dimensional 
factor analysis were applied. The research data were analyzed in computer with the package softwares 
of SPSS 18.0 and Lisrel 8.51. 
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INTRODUCTION 

In the past decade, rapid developments in information 
and communication technologies (ICT) have led to the 
developments in the field of education and increased the 
quality of education besides contributing to learning 
experiences (Yalman and Tunga, 2014). This rapid 
development of ICT has enriched teaching and learning 
experiences and led to a better-quality education (Telia, 
2011; Maldonado et al., 2011). Due to the rapid growth of 


the Internet, e-learning has become an alternative that 
has facilitated students’ learning (Wang Tzu-Hua, 2008). 
When compared with the traditional learning, it is seen 
that e-learning provides students with more sources and 
allows them to gain more satisfactory learning 
experiences via instructional activities (Saulnier et al., 
2008). Thanks to e-learning, students can carry out 
learning activities by determining the time and their pace 
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of learning the subjects. In other words, with the help of 
e-learning materials, students have the chance to 
evaluate their own advances. In face-to-face higher 
education, evaluation methods could be central as a 
basic component of effective learning and can define 
such methods in the learning process as measurement of 
students’ achievements and their pace of learning 
(Gikandi et al., 2011). Mid-term exams, end-of-term 
exams and quizzes may be given as examples of 
evaluation. In general, examination is one of the main 
methods used to confirm the results of students’ learning. 
Quizzes can be conducted to produce better instructional 
materials and for faculty members to evaluate students’ 
learning experiences. In this way, the quality of students’ 
academic achievements and their learning experiences 
can be increased (Miguel et al., 2016; Jordan, 2012; Levy 
and Ramim, 2007). 

It is known that different methods used to increase 
students’ academic achievements positively contribute to 
education levels. Probably, students’ attitudes constitute 
one of the most important processes that increase 
academic achievement (Gul et al., 2015). Attitude can be 
named as a mental process in which individuals 
determine their own behaviors in certain situations 
(Gagne, 1985). According to Smith (1968), attitude can 
be defined as a tendency that forms individuals’ thoughts, 
emotions and behaviors regarding a regular 
psychological object. An attitude and a learned fact 
shape individuals’ behaviors, can lead to a bias in the 
process of decision making. Attitudes occur as a result of 
the learning process and experiences (Tav§ancil, 2006). 
Online evaluations can be used in the evaluation of 
academic achievement. In addition, it contributes to 
meaningful learning and delivers the concepts or the 
learned information to the student as feedback. In 
traditional class applications, when teachers ask students 
a question, students are provided with a little chance of 
responding to the question. This makes it difficult for 
other students to see whether they have understood the 
subject or not. When questions are directed to students in 
distant education management system, the system can 
instantly provide online feedback and students thus, 
become more successful in learning when compared to 
the traditional system (Robles and Braathen, 2002). 
Online evaluation methods have a number of advantages 
mentioned above, yet there are limited areas for their 
application in higher education (Wen and Tsai, 2006). On 
the other hand, distance education at universities and 
private education institutions is still popular since it allows 
reaching a large population of students accommodating 
in different geographical regions. 

The number of distant education software is gradually 
increasing, and more students can now access these 
software. Preference of online test to evaluate students’ 
performances in educational processes will allow 
conducting both test applications and reporting 


procedures more rapidly. Attitudes are undoubtedly 
regarded as a good determiner of academic 
achievement. It is seen that technology has been in use 
to a great extent in educational studies in recent years. 
However, it is obvious that the number of studies 
conducted to reveal students’ attitudes towards online 
evaluations is quite limited (Dermo, 2009). In general, 
technology-aided studies were measured in relation to 
technology and computer, and their academic 
achievements were predicted by these variables. In 
addition, although, there are positive attitudes towards 
computers, students are likely to demonstrate negative 
attitudes towards online evaluation (Bindak and Qelik, 
2006; Ergun, 2002). 


Theoretical framework 

MOODLE has been used as an education platform via 
the Internet by a number of public institutions and private 
corporations for years. Many researchers investigated 
general system features of these web-based learning 
systems, users’ attitudes towards these systems and 
their levels of satisfaction with these systems (Coates et 
al., 2005; Engelbrecht, 2005; Marikar and Jayarathne, 
2016; Martfnez-Torres et al., 2008; Njenga and Fourie, 
2010; Seale and Cooper, 2010). There several reasons 
for this popularity of MOODLE: it is free of charge to 
access the system; it is an open-source system; and 
users can easily solve problems themselves (Gutierrez et 
al., 2010; Kakasevski et al., 2008; Limongelli et al., 2011; 
Xu and Mahenthiran, 2016). This education system, 
whose courses or curriculum can be designed in line with 
users’ needs via the web, have such basic features as 
homework and source sharing, questionnaires and 
forums as well as a testing system that allows measuring 
students’ success at the end of the education given. The 
study aimed at determining the views and attitudes of 
students taking education via the distance education 
management system towards the exam system. In this 
way, it could be easier to see whether the whole system 
functions well or not. 

Recent developments in information and 
communication technologies have caused a number of 
published papers to become out of date. In this respect, 
in web-aided instructional methods, information 
technologies are shaped in accordance with the 
renovations since these technologies are influenced by 
the related developments. In literature, when the papers 
in the field of web-aided education are examined, it is 
seen that issues mentioned by other studies were mostly 
examined via generalization (Brine et al., 2007; Georgouli 
et al., 2008; Zakaria and Daud, 2008). On the other hand, 
although, such systems are similar to each other in terms 
of function, methodology and form, their user interfaces, 
courses, questionnaires and exam systems could differ 
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(Brine et al., 2007; Romero et al., 2009). The basic 
features used here such as presentation of courses, 
questionnaires and forms are for general use of the 
system, while the exam system is used to measure and 
evaluate the success of students. Correct and valid 
measurement of students’ gains at the end of an 
education process depends on the features found in the 
exam module. Students’ evaluation of the exam module 
to determine its negative or positive aspects is important 
to reveal the related deficiencies. This study focused just 
on determining the attitudes towards the exam module 
used in MOODLE system rather than determining the 
attitudes towards exam modules used in web-aided 
education systems. 

Evaluation of the success of the web-based learning 
system of MOODLE, which has millions of users all over 
the world, will be possible via the related measurements 
and the results to be obtained from these measurements. 
When the related literature is examined, it is seen that 
there are studies which examined attitudes of users 
towards e-learning environments (Graf et al., 2009; 
Kakasevski et al., 2008; Sun et al., 2008) as well as 
those which investigate students’ views about the system 
(Kao and Tsai, 2009; Richardson, 2009; Sher, 2009; 
Yassine et al., 2016). Depending on the results of these 
studies, it could be stated that it will be better to 
overcome the problems existing or to exist in the system 
or to improve the popular applications in line with users’ 
feedbacks. 

Measurement and evaluation constitute the basis of 
education. All the evaluations to determine students’ 
levels of knowledge about the subjects they have been 
taught, should include objective and healthy evaluations. 
In contrast with the exams given in traditional education, 
those conducted in web-aided education include different 
features and norms; Questions should be clear and 
comprehensible; time limitations should focus on solving 
the questions; and a simple language should be used in 
the instructions to solve the questions. Here, the purpose 
is to minimize the probable problems. Students’ 
evaluations regarding the web-aided exam system at the 
end of the exams will help overcome related future 
problems. 

METHODS 

The participants in the study constituted those taking their theology 
undergraduate education via distance education system using the 
platform of MOODLE. In the study, a scale was developed to 
evaluate the students’ attitudes towards the web-based exam 
system (MOODLE). The data collected in the scale development 
process were analyzed with “Exploratory Factor Analysis”, “Uni- 
Dimensional Data Analysis” and “Two-Dimensional Factor Analysis” 
to examine appropriateness of the values to the fit indices. 

Sample 

The participants in the study were 1300 students who registered in 


Table 1 . Frequency and percentage distributions of the 
participants in terms of gender. 


Gender 

F 

% 

Female 

337 

51.61 

Male 

316 

48.39 

Total 

653 

100 


Theology Undergraduate Education Program; they received 
education with the e-learning management system in a distant 
education center of a state university. The scale designed for the 
study was applied via the system which was used to give 
education. A total of 1287 students responded to the scale 
conducted online. In this way, almost the whole research group was 
reached. In the study, as for the demographic backgrounds of the 
students filling out the scale in the study, 45.53% of them were 
female, and 54.47% of them were male. The data collected were 
examined in digital environment. Among the responses to the scale, 
those with all of its options marked same, those with most of its 
options marked same and those given to reverse items were 
examined, and the questionnaire forms which were not responded 
to correctly or completely were not included in the scope of the 
study to preserve the objectivity of the study. As a result, a total of 
653 questionnaire forms were included in the study. Table 1 below 
presents the frequencies and percentages of the participants with 
respect to their gender. According to the demographic backgrounds 
of the participants, 51.61% of them were females, and 48.39% of 
them were males (Table 1). 


Attitude scale for the web-based learning process 

For the purpose of determining the success, level and quality of 
education given via the web, several scales have been developed 
by researchers. Examining such scales in the study (Dermo, 2009; 
Gul et al., 2015; Bahar, 2014) and considering the standards 
previously determined (Swedish National Agency for Higher 
Education, 2008), a trail item pool was formed depending on the 
related literature, and faculty members expert in the field were 
asked for their views. The scale was presented to three experts in 
the field and to two Turkish language experts. The scale made up 
of 31 items, 23 of which were positive and eight of which were 
negative, was piloted to see whether it was comprehensible or not. 
The results of the statistical analysis of the collected data revealed 
that five items were difficult to comprehend and 10 items 
considered by the experts to be statistically inappropriate were 
excluded. As a result, there were 16 items in the finalized scale. 
Following this, the updated version of the scale was applied again 
to collect the research data. The scale included four items in the 
dimension of “System and Usability”, three items in the dimension 
of “Comprehensibility”, six items in the dimension of “Examination 
and Features” and three items in the dimension of “Security and 
Reliability”. The scale was designed as Five-point Likert-type with 
the choices of 1- I completely disagree, 2- I disagree, 3- I partly 
agree, 4- I agree and 5- I completely agree. In attitude scales, a 
five-point rating method ranging from “I completely agree” to “I 
completely disagree” could be used (Dunn-Rankin, 2004; 
Tav§ancil, 2005). When the related literature is examined, it is seen 
that scale development phases are as follows (Tav§ancil, 2005; 
Dunn-Rankin, 2004; Devellis, 2003; Karasar, 1995). 

1. Forming the item pool M 

2. Asking for experts’ view 
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Table 2. Fit indices and values. 


Fit Indices 

Criteria 

Value 

x 2 /sd 

<5/1 

3.69 

GFI 

>0.90 

0.94 

AGFI 

>0.90 

0.91 

RMSEA 

<0.08 

0.060 

S-RMR 

<0.05 

0.038 

CFI 

>0.90 

0.97 

NNFI 

>0.90 

0.96 

IFI 

>0.90 

0.97 


3. Conducting the pilot application 

4. Applying the draft scale to the study group and factor analyses 

5. Calculating the reliability of the scale 

In order to determine the factor loads predicted for scale 
development, exploratory factor analysis (EFA) and confirmatory 
factor analysis (CFA) were conducted. After the results of these two 
analyses that were obtained, confirmatory factor analysis was used 
for the model data fit. The fit indices used in the study were Chi- 
Square fit test, Goodness of Fit Index (GFI), Adjusted Goodness of 
Fit Index (AGFI), Root Mean Square (RMS) and Root Mean Square 
Error of Approximation (RMSEA). Table 2 presents the results 
obtained via the analysis of the data. The value of x 2 /sd obtained in 
the study was lower than 3, which demonstrated that the model had 
an acceptable fit (§im§ek, 2007). The value in question was higher 
than three. As the value of x 2 /sd is sensitive to the size of the 
sample, it should be interpreted with other fit indices (Joreskog and 
Sorbom, 1999). According to the model data fit, the value of 0.94 
GFI was higher than 0.09; the value of 0.91 AGFI was higher than 
0.09; the value of 0.064 RMSEA was lower than 0.08; and the value 
of 0.038 RMR was lower than 0.05. 


FINDINGS 

Exploratory factor analysis 

The scale developed was applied to 1288 individuals, yet 
the analysis was carried out with 653 participants due to 
the fact that some of the questionnaires were not filled 
completely or some of them included items all with the 
same options marked. The Kaiser-Mayer-Olkin (KMO) 
value of the scale was 0.877, and Barlett’s test was found 
significant (p < 0.01). The scale included a total of 16 
items and four factors. In line with the expert view, no 
change was done in the scale. Table 3 presents the 
results of the exploratory factor analysis. 


Uni-dimensional confirmatory factor analysis (CFA) 

Figure 1 presents the results of the confirmatory factor 
analysis conducted to determine the fit between the 
factors and items in the scale. According to the results of 
the confirmatory factor analysis, the Chi-Square value of 


x 2(98,N=685)=yggg gg ca | cu | a t ec | f or the model-data fit was 
significant (p < 0.000). The fit statistics values calculated 
using the Lisrel GFI = 0.94, AGFI = 0.91, CFI = 0.97, 
NNFI = 0.96 software were as follows: RMSEA = 0.064, 
RMR = 0.038, and = 0.97 dir. Since these values were in 
appropriate ranges, there was no need for any 
modification in the scale. 

IFI 

Two-dimensional confirmatory factor analysis (CFA) 

The attitude scale for the web-based learning process 
with its 16 items and four factors was tested with two- 
dimensional confirmatory factor analysis (CFA). 
According to EFA, the items for the fact of System and 
usability had standard solutions of 0.64, 0.66, 0.58 and 
0.51; those for the fact of Examination and Features had 
standard solutions of 0.49, 0.49, 0.54, 0.56, 0.54 and 
0.53; those for the factor of Comprehensibility had 
standard solutions of 0.75, 0.71 and 0.90; and those for 
the factor of Security and Reliability had standard 
solutions of 0.53, 0.54 and 0.49. As can be seen, all the 
standard solutions were found to be higher than 0.45. As 
a result of CFA, the fit statistics values calculated as x 2 /sd 
= 3.36 were as follows: RMSEA = 0.064, RMR = 0.038, 
GFI = 0.94, AGFI = 0.91, CFI = 0.97, NNFI = 0.96 and IFI 
= 0.97. All the fit indices obtained were found to 
demonstrate an acceptable fit according to Schermelleh- 
Engel, Moosbrugger and Muller (2003) (Figure 2). It was 
seen that the values obtained regarding the items and the 
whole scale were in acceptable ranges for the 
applicability of the scale. 


Results of reliability analysis 

Table 4 presents the reliability coefficients for each factor 
in the scale. The Cronbach Alpha value for the whole 
scale was calculated as 0.873. The Cronbach Alpha 
values were calculated as 0.834 for the sub-factor of 
“System and Availability”, as 0.793 for the factor 
“Comprehensibility”, as 0.761 for the factor of 
“Examination and Features” and as 0.740 for the factor of 
“Safety and Reliability”. The Cronbach’s alpha values 
obtained in relation to the sub-factors demonstrate that 
the scale developed was valid (Brownlow, 2004). 


DISCUSSIONS 

Depending on the related need in literature, the present 
study aimed at developing a valid and reliable attitude 
scale to evaluate the attitudes of distant education 
theology undergraduate students towards the Web- 
Based Examination System (Appendix). For this purpose, 
in line with the scale development phases reported in 
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Table 3. Exploratory factor analysis for “web-based learning attitude scale”. 

Item number 

System and usability Comprehensibility Examination and features 

Security and reliability 

Ml 

0.781 



M2 

0.813 



M3 

0.816 



M4 

0.708 



M14 

0.840 



M15 

0.831 



M16 

0.661 



M8 


0.524 


M9 


0.513 


M10 


0.551 


Mil 


0.582 


M12 


0.792 


M13 


0.696 


M5 



0.706 

M6 



0.772 

M7 



0.757 



Figure 1. Confirmatory Factor Analysis Results for “Web-Based Learning Attitude Scale”. 
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Chi-Square= : 329.23, df=98, P- 1 ralue=G.OOOGO, RMSEA=Q.060 


Figure 2. Two-dimensional confirmatory factor analysis (CFA) Results for "Web-Based 
Learning Attitude Scale”. 


Table 4. Cronbach’s alpha values regarding the sub-factors of the web-based learning attitude 
scale. 


Factors 

Number of Items 

Reliability Coefficient (a) 

System and Availability 

4 

0.834 

Comprehensibility 

3 

0.793 

Examination and Features 

6 

0.761 

Safety and Reliability 

3 

0.740 


related literature (Devellis, 2003; Dunn-Rankin, 2004; 
Karasar, 1995; Tav§ancil, 2005), an item pool was 
formed; experts’ views were taken regarding the items; 
and the trial item pool was piloted. Following the pilot 
application, the draft scale was applied to the study 
group, and the data were examined in terms of the 
distribution of the scale scores obtained before 
conducting the Exploratory Factor Analysis. After the 
distribution was found to be at the desired level, to 
determine the factor structure of the scale, basic 
components analysis was preferred as the factorization 
method, and the maximum varimax technique, one of the 


vertical rotation method, was preferred considering and 
significance. In the study, Exploratory Factor Analysis 
was conducted sequentially and gradually on the data 
several times. For the statistical evaluation of the data 
collected during the application, five items lacking unity of 
meaning and ten items considered statistically to be 
inappropriate were excluded from the scope of the study. 

After the scale items were updated, the scale was 
applied again to collect the research data. The final 
version of the scale, which was initially made up of 31 
items, included 16 items. In addition, factor analysis 
applied again to the final version of the scale. According 
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to the results of the exploratory factor analysis, there 
were four factors with an Eigen value higher than 1. It 
was found that the first factor (System and Availability) 
contributed to the common variance with a rate of 
17,154%; the second factor (Comprehensibility) with a 
rate of 13,834%; the third factor (Examination and 
Features) with a rate of 16,496%; and the fourth factor 
(Safety and Reliability) contributed to the common 
variance with a rate of 12,868%. The contribution of the 
four factors to the total variance was calculated as 
60,353%. Also, the Kaiser-Mayer-Olkin (KMO) value of 
the scale was calculated as 0.877, and Barlett’s test was 
found significant (p < 0.01). In order to evaluate the 
validity of the two-factor structure obtained as a result of 
the exploratory factor analysis, confirmatory factor 
analysis (CFA) was conducted. When the CFA fit values 
were examined, the Chi-Square value of x 2 (98,N=685) = 
51.38 calculated for the model-data fit was found 
significant (p <0.000). The fit statistics values calculated 
as a result of the analysis carried out with Lisrel software 
were as follows: RMSEA = 0.064, RMR = 0.038, GFI = 
0.94, AGFI = 0.91, CFI = 0.97, NNFI = 0.96 and IFI = 
0.97. Since the values obtained were in appropriate 
ranges, no modification was done in the scale. 

The web-based learning attitude scale made up of 16 
items and four factors was also tested with the two- 
dimensional confirmatory factor analysis (CFA). The fit 
statistics values found to be x2/sd = 3.36 as a result of 
CFA were as follows: RMSEA = 0.064, RMR = 0.038, 
GFI = 0.94, AGFI = 0.91, CFI = 0.97, NNFI = 0.96 and IFI 
= 0.97. According to Schermelleh-Engel, Moosbrugger 
and Muller (2003), all the fit indices obtained were found 
to demonstrate acceptable fit. In addition, the Cronbach 
Alpha values were calculated as, 834; for the factor of 
“System and Availability”, as, 793; for the factor of 
“Comprehensibility”, as, 761; for the factor of 
“Examination and Features” and as, 740 for the factor of 
“Safety and Reliability”. The scale could be used as a 
valid and reliable data collection tool in studies to be 
conducted not only to determine the attitudes of students 
from the Department of Theology Undergraduate 
Education Program towards web-based distant education 
as well as towards related exam practices but also to 
examine the factors influential on their attitudes. 

Conclusion 

Rapid changes and developments in computer and 
Internet technologies lead to the development and spread 
of e-learning management systems. In this process of 
changes, e-evaluation has gradually gained more 
importance and become a significant part of this 
transformation. Evaluation on paper-pen basis is quite an 
inefficient method of evaluating students’ success and 
making education decisions (Sirakaya et al., 2014). 
Although, the reliability and objectivity of exams 


conducted via e-learning platforms are examined by 
researchers and experts, these exams have a number of 
advantages when compared to those conducted with 
traditional methods. The most prominent advantages of 
e-evaluation include saving time and cost, gathering the 
responses to the questions in computer environment, 
providing appropriate and rapid feedback, allowing 
flexibility, increasing the reliability by minimizing the 
mistakes made by human (Struyven, Dochy and 
Janssens 2002; Angus and Watson 2009), decreasing 
faculty members’ involvement (Anderson et al., 2005) 
and obtaining the results rapidly (Kuhtman, 2004). In 
addition, exams conducted in computer environment, in 
contrast with paper-pen exams, allow enriching the 
presentation of information via integration of multimedia 
elements (Liu et al., 2001). Obviously, electronic 
evaluations are beneficial. Faculty members can obtain 
rapid results, thanks to e-evaluation, and use of 
computers that will make education better. This will allow 
transition from a traditional education environment to the 
one that makes students more active and contributes to 
their learning. In recent years, the traditional 
measurement and evaluation methods in the field of 
education have been replaced by e-evalaution. Thanks to 
this, the field of measurement and evaluation has gained 
a new dimension. As an alternative, faculty members try 
to spread the use of the Internet and computer in the field 
of measurement and evaluation (Ozmen, 2006; Lawrenz 
et al., 2001). 
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Appendix: Attitude Scale towards Web-Based Examination System (MOODLE) 
Part 1 . Demographics 


Sex: 

Class: 


Female 
3 


Male 

4 


How long have you been using a computer? 


Less than a year 


1 -2 years 


2-4 years 4-6 year 


What is your level of knowledge of computer use? 




None 

Little 

Average 


Good 



How often do you u se th e Internet? 


Once a day 


Twice or 3 times a day 


Twice or 3 times a week 


J more than 6 years 
Very good 

1 

Once a week Once a month 


Did you ever use a learning management system (distance education management system) in your previous education 
life or in a course? 

Yes I No 


Part 2. Scale 
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1 Web-based evaluation has an important role in higher education. 

2 Web-based evaluation is appropriate to religious education. 

3 Web-based evaluation could contribute to my learning. 

4 Web-based evaluation has an important role in distance learning (Moodle). 

5 Web-based evaluation is as reliable as paper-based evaluation. 

6 Web-based evaluation is reliable in terms of exam results. 

7 I prefer web-based exams to traditional exams. 

8 Feedback provided via the exams conducted over Moodle (exam module used) 
contributes to my learning. 

9 I found use of the Moodle web-based exam module easy. 

10 Web-based exams have more advantages when compared to traditional exam 
environment. 

11 Web-based exam module could be suggested for those who favor traditional exams. 

12 Seeing the wrong answers given to questions in web-based exams contribute 
significantly to overcoming inefficacies. 

13 Seeing each question directed in web-based exams on the screen is an advantage. 

14 Questions directed in web-based exam module are legible when compared to those 
directed in traditional exams. 

15 Questions directed in web-based exam module are more comprehensible. 

16 Web-based exam module is as successful as the traditional exam system conducted 
in class environment. 















